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ABSTRACT 



A method and apparatus for transmitting encoded data from 
one computer system to another using an AT command set 
compatible communication device is described. Data being 
sent from a communications application to the AT command 
set compatible communication device is monitored, and 
encryption is performed in response to detecting a connect 
message and a carrier detect signal received from the com- 
munication device and changing in response to said detec- 
tion the state of an encryption application from a first state 
to a remote state and the data is encrypted using a stream 
encryption scheme before it is passed to the communication 
device. When a break sequence is detected or a carrier detect 
signal is deasserted, encryption is ceased thus having the 
encryption application automatically return to the first state. 

21 Claims, 7 Drawing Sheets 
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MODEM COMPATIBLE METHOD AND 
APPARATUS FOR ENCRYPTING DATA 
THAT IS TRANSPARENT TO SOFTWARE 
APPLICATIONS 

s 

This is a Continuation Application of application Ser. No. 
08/291,160, filed Aug. 16, 1994 now abandoned. 

BACKGROUND OF THE USTVENTIGN 

1. Field of the Invention 

The described invention relates generally to the field of 
data communications. More particularly, the invention 
relates to a method and apparatus for encrypting data that 
can be used with standard software communication appli- 15 
cations using modems. 

2. Prior Art 

The use of a modem in combination with a standard 
telephone line is a common method for exchanging data 
between computer systems. Modems use a standard com- ^0 
munication interface that allows modems from different 
manufacturers to communicate with each other. The 
"Advanced Technology" command set has also become a 
defacto standard command language with which computer 
systems can control a modem. This combination of a stan- ^5 
dard command language and a standard interface allows 
modcntis made by different manufacturers to provide com- 
munication capability with computer systems which are also 
from different manufacturers. This compatibility gives a user 
the ability to purchase computers and modems based on 
price and quality considerations only, and therefore is desir- 
able. 

Most recently, modems have been developed which oper- 
ate over cellular networks using standard cellular phones. In 
addition, AT compatible modems have been developed that "'^ 
operate over the radio network. 

The use of a modem with a highly compatible interface 
creates some security risk, however, because anyone able to 
detect the signals used by the modems to exchange infor- 
mation can determine the contents of that information. In the 
past, this security risk has been minimized by the relatively 
high degree of privacy offered by a standard phone line. The 
use of cellular telephones as a substitute for traditional 
telephones, however, has removed this at least somewhat 
satisfactory degree of security because the signals carrying 
the information being exchanged are broadcast where they 
can be detected by anyone so inclined. Thus, the use of a 
cellular telephone for data communication between com- 
puter systems has created the need to make the information 
transmitted by the cellular telephone unreadable to anyone 
other than the intended recipient. 

The growing presence of mobile computing involving 
portable computers and wireless data communications poses 
a considerable threat to corporate security since wireless 55 
communications are vulnerable to interceptions. The com- 
munications software used with these portables relies on 
simple passwords to protect access to the network and the 
login script generally holds the password in plain text. In 
other words, the stolen portables gives the thief an easy 
access to the network. Even some applications that use 
encryption do not employ a strong encryption scheme. 

Encryption provides one method for making the informa- 
tion transmitted by modems unreadable to anyone other than 
the intended recipient. While modems that provide encryp- 65 
tion capability exist, the encryption feature cannot be uti- 
lized unless the receiving modem also possesses the same 
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encryption capability. This requirement eliminates much of 
the desired compatibility previously offered by modems that 
did not posses encryption capability. Additionally, this 
hardware-based solution adds to the cost of the modem. 
Alternatively, communication software programs may incor- 
porate an encryption feature, however, this requires the two 
computer systems communicating with each other to utilize 
communication software applications that incorporate the 
same encryption method. Once again, this reduces users 
choices and eliminates the previously established compat- 
ibility among computer systems. Thus, it is desirable to have 
a method and apparatus of encrypting data that allows 
various standard computer systems, software applications, 
and modems to be used interchangeably and which can be 
uniformly applied vAth modems that operate over cellular 
networks, radio networks and regular telephone lines. 

BRIEF SUMMARY OF THE INVENTION 

The present invention pertains to an apparatus and method 
for data encryption for a computer system communicating 
via a modem. When a communication application running 
on the computer system attempts to write data to the modem, 
the transaction is intercepted and the data is observed. If the 
data contains commands being issued to the modem, the data 
is passed onto the modem directly without encryption. 
Otherwise the data is encrypted before being sent to the 
modem. The modem then transmits the data in a normal 
fashion to a remote modem to which it is connected. 
Immediately after the connection is established with the 
remote modem, an encryption key for a session is generated 
randomly and securely transmitted to the remote end using 
the RSA public-key cryptography technology. When the 
remote modem receiving the encrypted data attempts to 
provide the encrypted data to a remote software application, 
this transaction is also intercepted and the data is decrypted 
before being provided to the remote software application. 
The encryption scheme works transparent to the application 
and requires no changes to the existing applications. 

In accordance with one aspect of the invention, an 
encrypted data stream is generated by performing an 
exclusive-or operation on an infonnation data stream and an 
encryption data stream. The encryption data stream is gen- 
erated by performing a Data Encryption Standard ("DES") 
encryption operation on the encrypted data stream utilizing 
an encryption key. A special startup sequence is utilized 
while new data for the encryption data stream is being 
generated. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is an illustration of two computer systems com- 
municating in accordance with one embodiment of the 
invention. 

FIG. 2 is a block diagram illustrating a computer system 
configured in accordance with one embodiment of the 
invention. 

FIG, 3 is a block diagram that illustrates the interaction of 
the various systems of the computer systems shown in FIG. 
1 when configured in accordance with one embodiment of 
the invention. 

FIG. 4 is a block diagram illustrating the modules that 
make up TSR's 301 and 305 shown in FIG. 3. 

FIG. 5 is a stale diagram illustrating the state transitions 
of TSR's 301 and 305 shown in FIG. 3. 

FIG. 6 is a table illustrating the state transitions experi- 
enced by INT 14 redirector 402 in response to the detection 
of a connect message. 
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FIG. la and lb are flowcharts that illustrate the logic used 
to implement the default break sequence detection function. 

FIG. 8 is a block diagram illustrating the screen encryp- 
tion and decryption method incorporated by stream DES 
engine 404 in one embodiment of the invention. s 

DETAILED DESCRIPTION OF THE 
INVENmON 

An apparatus and method for encrypting data transmitted 
by a computer system through a modem is described in jq 
detail. In the following description for purposes of 
explanation, specific details such as interrupt configurations, 
hardware components, communication devices including AT 
mobidems, etc., are set forth in order to provide a thorough 
understanding of the present invention. However, it will be 
apparent to one skilled in the art that the present invention 
may be practiced without these specific details including the 
use of standard modems. In other instances, well known 
structures, devices, functions, and procedures are shown in 
block diagram form in order to avoid obscuring the present 
invention. It should be noted that the present invention can 
be applied to a variety of different processor architectures. 
Furthermore, the various systems that make up the invention 
can be practiced in a variety of manners such as through the 
use of hardware and software encoding mechanisms. 

Referring now to FIG. 1, two computer systems 10 and 12 
communicating via AT mobidems 18 and 20. AT mobidems 
18 and 20 are modems that utilize the AT command set 
combined with radio transmitter-receivers. While the 
modem function and the transmitter-receiver function are 30 
shown incorporated into a single system, the two functions 
could be performed by two separate systems. The radio 
signals generated by AT mobidem 18 in response to digital 
signals from the computer system 10 are transmitted to the 
nearest base station 22 of the radio network. The base station 35 
22 transmits the information to the radio network which in 
turn sends the information to the base station 21 closest to 
the remote mobidem 20. The remote mobidem 20 receives 
the signals from the base station 21 and, converts the signals 
back into digital signals that can be processed by computer 4q 
system 12. As the radio signals propagate from the mobidem 
to the base station 22 and from the base station 22 to the 
radio network, the radio signals can be monitored and 
understood by anyone who possesses the proper equipment. 

As with standard modems, AT mobidems 18 and 20 are in 45 
a command mode when initially activated. In the conmaand 
mode, the mobidems are controlled via the AT command set 
of instructions. Further in the command mode, all data 
received by AT mobidems 18 and 20 is interpreted as 
commands and can cause the mobidems to change states and 50 
to perform certain operations, such as dialing a phone 
number. Once a connection is established between AT mobi- 
dems 18 and 20, the mobidems enter data mode. When in 
data mode, AT mobidems 18 and 20 exchange data sent to 
them from computer systems 10 and 12 respectively. Data ss 
received from the other AT mobidem is provided by AT 
mobidems 18 and 20 to the computer systems to which they 
are connected. Unless a special "break" sequence of data is 
provided within a predetermined set of time intervals, the 
data received from computer systems 10 and 12 will not alter 60 
the state of AT mobidems 18 and 20, or cause them to leave 
data mode. Additionally, AT mobidems 18 and 20 both assert 
a carrier detect signal (CD) when in data mode as per the AT 
command set protocol, indicating that the mobidems arc in 
communication with another modem or mobidem. 65 

FIG. 2 is a block diagram of a computer system config- 
ured in accordance with one embodiment of the invention. 
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In this embodiment, application programs and "terminate 
and stay resident" ("TSR") programs are stored on disk 
drive 200. These programs are loaded into DRAM memory 
201 by microprocessor 202 once the computer is activated 
and the programs are run. TSR programs are utilized in the 
DOS® operating system environment and remain in 
memory while application programs are running. TSR pro- 
grams detect when certain events such as an input/output 
transaction take place within the computer system and then 
take predetermined action in response. Other operating 
systems including Macintosh® System 7® software allow 
similar "background" software to be incorporated into their 
system. Microprocessor 202 receives instructions from the 
appUcation programs located in DRAM 201 which, on 
occasion, cause microprocessor 202 to perform read and 
write operations to input/output ("I/O'') device 203 through 
system bus 204. 

FIG. 3 is a block diagram illustrating the interaction of the 
various systems within computer systems 10 and 12 (FIG. 1) 
in communication with each other in accordance with one 
embodiment of the invention. In this embodiment, commu- 
nication application 300 is a CC: Mail program from Lotus 
Development Corp., although other communication appU- 
cation programs such as terminal emulators or file transfer 
programs can be utilized. Communication application 300 
communicates with input/output ("I/O") buffer 302 through 
TSR 301. Data passed to I/O buffer 302 is sent to AT 
mobidem 18 which is coupled via radio link to AT mobidem 
20. AT mobidem 18 transmits the data to AT mobidem 20 via 
the cellular radio link, which passes the data to communi- 
cation application 306 via I/O buffer 307 and TSR 305. 

When configured in accordance with one embodiment of 
the invention, communication application 300 utilizes the 
INT 14 Application Program Interface ("API") call in an 
interrupt mode, as opposed to a polling mode, to perform 
input/output operations within computer system 10 (FIG. 1). 
The INT 14 API is a procedure within the DOS operating 
system environment to write or read data to or from one of 
the I/O ports. The four INT 14 API requests are as follows: 
INT 14 AH-0, DX=Port which initializes the specified COM 
port; INT 14 AH«1, AL=Char, DX=Port which writes a 
character to the specified COM port; INT 14 AH=2, 
AL=Char, DX=Port which reads a character from the speci- 
fied COM port; and INT 14 AH=3, AL-Char, DX=Port 
which reads the COM port status. While there are other 
procedures and methods for performing I/O operations that 
could be utilized in other embodiments of the invention, the 
INT 14 API is the most commonly used and therefore is 
desirable. Other operating systems may have different meth- 
ods for performing I/O operations that can be accommo- 
dated by a different embodiment of the invention including 
the DLLintermpt scheme in Microsoft™ Windows^^". 

When computer system 10 (FIG. 1) is configured in 
accordance with one embodiment of the invention, TSR 301 
is placed in memory while communication application 300 
is running. One method to accomplish this is to activate 
communication application 300 through a batch (.BAT) file 
that also activates TSR 301, and that de-activatcs TSR 301 
once communication application 300 is de-activated. When 
communication application 300 attempts to conduct an I/O 
operation using the INT 14 API call, TSR 301 intermpts the 
operation and examines the data being transferred. TSR 301 
monitors the data being sent between computer system 10 
and AT mobidem 18 and changes state depending on the data 
detected. If a connection has been established with a remote 
modem, TSR 301 encrypts the data before passing the data 
on to I/O buffer 302. If no connection has been established, 
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TSR 301 passes the data on without encryption to I/O buffer CONNECr_RESP variable is set to True and the CD signal 

302 so that communication application 300 may control AT is asserted, INT 14 redirector 402 enters encrypt mode 501 

mobidem 18. where it directs all data it receives from either communica- 

FIG. 4 is a block diagram illustrating the three main tion application 300 or AT mobidem 18 to stream DES 

modules that makeup TSR 301 (FIG. 3). The three modules s engine 404. Stream DES engine decrypts the data received 

are COMM driver 400, INT 14 redirector 402, and stream from AT mobidem 18 and encrypts data from communica- 

DES engine 404. COMM dnver 400 is an interrupt driven tion application 300 before returning the data to INT 14 

serial port driver generaUy consistent with other drivers redirector 402. In addition to encrypting and decrypting 

known in the art hat provides a .simple procedural API for ^^ile in encrypt mode 501, INT 14 redirector 402 tracks the 

setting up a COM port with the necessary communicalion ,o jata sent from communication application 300 for the status 

parameters and mterrupt haMmg, as well as reading and ^^^^ cD signal. If the CD signal is de-asserted, or the 

writing characters from the COM port. Additionally, COMM ^p^^.; ^^^^ sequence of data is detected from communi- 

dnver 400 allows the normal mtermp^^^ nation application 300, INT 14 redirector 402 enters 

and the COM port to be released. INT 14 redirector 402 no-encrypt mode 500 

receives data from COMM driver 400 and communication r-T^ ^ - ti •^^ . . .. , 

application 300 and either directs that data through DES ^ ^^^^ illustratmg the state transitions that 

engine 404 orpasses the data on depending on whether INT f^lTP'^J- ^f^T"" "". .u ^^^^^ 

14 redirector 402 is m an encrypt or a no-encrypt mode. DES l^^^'''' to set the vanable CONNECT RESP 

engine 404 encrypts and decrypts the data received from ° T^^* commands are issued to AT mobidem 18 such 

lOT 14 redirector 402 commumcation is established with AT mobidem 20, the 

1, • . . %u *u J • 1 ■ AT command set requires the AT mobidem 18 to provide a 

While generally consistent with other dnvers known in ^ *u ♦ • ^ ^ ^ ti 

*u ^ Ar\\jtx4 A • At\i\ 1 • 1 J ji 1 connect result code that is configured as follows: 

the art, COMM dnver 400 employs a special deadlock uw\„nr\KrKmr^*^\ \ » u • j- * 

, . u- u • /t u *- -^u \r\nCONNECT*\r\n , where indicates one or more 

prevention mechanism which is useful when operating with . , u i <\ > * *u r r j u 

^ . . r>r> \/f ATM , • characters. The symbol *\r represents the line feed character 

communications software such as CC: Mail™ that ignores a a , * #t, ♦ u * tt. 

• .7 J J u J . the symbol *\n represents the return character. The 

No earner message provided by AT command set com- nc . . „ , j * * ■ i 

... ' J • u ^ characters that usually correspond to the astensk are num- 

paiible communications devices when a connection with tk,* a ftu . ur u j a 

• * • J • • u 1 vr >^ . bers that indicate the speed of the connection established. An 
another commumcation device is broken. The No Carrier 

message indicates that the connection with another commu- 
nications device is lost and that AT mobidem 18 is entering 

command mode. While TSR 301 will recognize this and 30 S^ill referring to FIG. 6, when INT 14 redirector 402 is 

change state accordingly as described below, communica- activated, it is in state 0. As the first *\r' character is 

lion application 300 may not, and may continue to send data detected from ATM mobidem 18 (FIG. 1), INT 14 redirector 

to AT mobidem 18. If the data sent to AT mobidem 18 402 enters state 1. When the *\n' character is detected INT 

contains the character sequence that places AT mobidem 18 14 redirector 402 enters state 2. As the 'C\ '0\ and *N' 

in echo mode, AT mobidem 18 will echo all data sent to it. 35 characters are detected, INT 14 redirector 402 enters states 

This can cause an RTS (ready/request to send) signal that is 3, 4 and 5 respectively Should any of the characters received 

asserted by computer system 10 when it is ready to receive not be part of the expected connect message, INT 14 

data from AT mobidem 18 to be de-asserted when the input redirector 402 returns to the default state of 0. Once INT 14 

queue is full. The de-assertion of the RTS signal, in turn, redirector 402 reaches state 9 the default state also becomes 

causes AT mobidem 18 to de-assert a CTS (clear to send) 40 state 9, and INT 14 redirector 402 remains in this state while 

signal used to indicate the RTS signal's readiness to receive the unknown characters that indicate the speed of the 

data, creating a deadlock condition when COMM driver 400 connection, and possibly other information, are received, 

reaches the transmit queue full condition. To prevent this When the final *\r' and '\n' characters are detected, INT 14 

deadlock, COMM driver 400 maintains the assertion of RTS redirector 402 enters state 10, and then sets CONNECT„ 

when AT mobidem 18 is in command mode. The assertion 45 RESP to True. When the CONNECT_RESP variable is 

of RTS causes AT mobidem 18 to maintain assertion of CTS, Tnie, and the CD signal is asserted, INT 14 redirector 402 

thus allowing all data sent by communication application transitions from no-encrypt mode 500 to encrypt mode 501. 

300 to be written to AT mobidem 18 preventing the output FIG. la is a flow chart illustrating a procedure called 

queue from filling up. This prevents the deadlock situation MonitorPIus implemented by INT 14 redirector 402. Moni- 

discussed above. 50 torPlus detects a break sequence issued from computer 

FIG. 5 is a state diagram illustrating the state transitions system 10 used to force AT mobidem 18 into command 
of INT 14 redirector 402 when configured in accordance mode. When this sequence is detected MonitorPIus causes 
with one embodiment of the invention. When INT 14 INT 14 redirector 402 to switch states, and sends on the 
redirector 402 is activated, INT 14 redirector 402 is placed unencrypted break sequence to AT mobidem 18 using the 
in the "no-encrypt" mode 500. In no-encrypt mode 500, INT 55 procedure switchFSM. When INT 14 redirector 402 is first 
14 redirector 402 traps each INT 14 API input/output activated, it assumes that AT mobidem 18 is utilizing the 
operation between communication application 300 and AT default break sequence of a one pause (guard time interval) 
mobidem 18, tracks each byte of data sent and received, and followed by three *+' characters and another one pause 
passes the data on unencrypted. When AT mobidem 18 (prc-post-guard time interval). This configuration may be 
establishes a connection with AT mobidem 20, either 60 changed by the user, however, by indicating a new default 
through commands issued by communication apphcation time and character set to TSR 301 and to AT mobidem 18. 
300 or through a call received by the AT mobidem 18 from The MonitorPIus procedure tracks each character sent by 
AT mobidem 20, AT mobidem 18 issues a "connect" mes- communication application 300 to AT mobidem 18. For each 
sage and assert the carrier detect ("CD") signal in accor- character received MonitorPIus first determines if the pro- 
dance with the AT command set protocol. 65 cedure switchFSM is ninning at step 700 indicating that a 

When INT 14 redirector 402 detects the connect message, full escape sequence has previously been detected and that 

it sets the variable CONNECT_RESP to True. Once the AT mobidem 18 is being placed in command mode. If 
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switchFSM is not running, MonitorPliis determines at step 
702 if something other than a plus character is being sent and 
the variable PlusCount is not equal to zero. If both of these 
conditions are true, any previously encountered plus char- 
acters are flushed at step 704 and a "forward char and return" 
operation is performed at step 706. Otherwise, MonitorPlus 
determines at step 708 if something other than a plus 
character is being transmitted and, if not, a simple forward 
char and return operation is performed at step 706. If at step 
708 it is determined that plus character is being transmitted 
and it is then determined at step 710 that PlusCount is zero. 
The procedure determines at step 712 if the present time 
interval is greater than the guard time. If the present time 
interval is greater than the guard time, then the PlusCount is 
incremented at step 714 and the plus character is stored. The 
procedure then returns to step 716. If the plus count is not 
zero, the procedure determines at step 718 if the present time 
interval is less than or equal to the guard time. If the present 
time interval is greater than the guard time, than the proce- 
dure flushes the stored plus characters and performs a 
forward char and return operation. If the time interval is less 
than or equal to the guard time at step 722, the plus count is 
incremented at step 720. If the plus count equals three, 
switchFSM is initiated at step 724. Otherwise, the plus 
character is stored and the program returns. 

FIG. Ih is a flow chart illustrating the logic implemented 
by sub -routine switchFSM. SwitchFSM places INT 14 redi- 
rector 402 in the no-encrypt mode and sends an unencrypted 
break sequence to AT mobidem 18 when called. When 
SwitchFSM is called, if at step 750 the variable FSMstate is 
in the INIT state, SwitchFSM places FSMstate in the Wait 
Guard Time state at step 752 and waits the pre -guard time 
delay of 1 second at step 754 and returns to step 750, Since 
the result of the test of step 750 will now be negative, step 
756 is performed. If at step 756 FSMstate is in the Wait 
Guard Time state, three characters are sent. In addition 
FSMstate is set to Sent 3 Plus at step 758, and the program 
wails the post guard time of 2.0 second during step 759. 
Otherwise, step 760 is performed. If at step 760 FSMstate is 
in the Sent 3 Plus state, FSMstate is set to the INIT and INT 
14 Redirector of TSR 301 is set to no-encrypt mode 500 at 
step 762. Then the program returns. This simultaneously 
changes the slate of INT 14 redirector 402 to no-encrypt 
mode and places AT mobidem 18 in the command mode so 
that the commands issued by communication appUcation 
300 are processed by AT mobidem 18. 

FIG. 8 is a block diagram illustrating the stream encryp- 
tion and decryption method incorporated in stream DES 
Engine 404 (FIG. 4) in one embodiment of the invention 
when INT 14 redirector 402 (FIG. 4) is in encrypt mode 501 
(FIG. 5), The algorithm takes an 8-bit plaintext character and 
outputs an 8-bit ciphertext character making it a perfect fit 
for encryption over serial transmission lines. 

Both the transmitter and receiver has a 64-bit input 
register (11-18) 804 and 806) respectively, which operates as 
a shift register and a 64-bit output register (01-08) 802 and 
807 respectively. When a plaintext character arrives it is 
EXCLUSIVE-ORed 812 with 8-bits of the output register, 
01 801. Output registers 02 thnDUgh 08 are never used. The 
character 803 thus created is both transmitted to the receiver 
and shifted into the input register, pushing 18 ofif the end. 
Then DES is computed on the new 64-bit input register 804 
and the result is stored into the 64-bit output register 802, 
making it ready for the next input. 

At the receiving end, the incoming character 803 is first 
EXCLUSIVE-ORed 814 with byte 01 in register 807 
(yielding the plaintext), and then shifted into byte II in 
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register 806. If the sender and receiver start out with 
identical input registers, they remain identical forever. In 
other words byte 01 at the transmitter register 802 will 
always be the same as byte 01 at the receiver register 807. 

S Since the incoming plaintext character at the transmitter is 
being EXCLUSIVE-ORed with the same character as the 
incoming ciphertext character at the receiver, the output at 
the receiver (byte stream 800) is the original plaintext. 

The important property of this stream encryption algo- 
rithm is that 01 depends on the entire history of the 
plaintext, so that a repeated pattern in the plaintext wiU not 
generate a repeated pattern in the ciphertext. Also, it is 
important to note that DES is operated in encryption modes 
at both the transmitter and the receiver. 

J 5 The DES module 811 initiahzes the input shift registers 
with known input values 809 at the time the connection is 
established between the two modems. That is, whether the 
Carrier Detect is asserted and the Modem Connect result 
message is received, the DES is initialized. 

20 The DES stream encryption method is employed because 
it provides high security without requiring extensive com- 
puter resources. Alternative embodiments of the invention 
may incorporate other symmetric key stream based encryp- 
tion methods including the RC4™ encryption scheme 

25 known in the art. These alternative embodiments, which are 
part of the implementation of the invention, could also allow 
one to switch between these various encryption methods to 
ensure maximum repeatability. However, by their very 
nature, the symmetric key encryption schemes require that a 

3Q key be established before communication is established. In 
an alternative embodiment, the symmetric encryption key 
can be generated after the connection is established between 
AT mobidem 18 and 20. This is accomplished using a 
pubhc-private key encryption method well known in the art, 

35 such as the RS A encryption scheme, that allows messages to 
be encrypted using a public key that only allows for one way 
encryption. When the public-key method is employed, INT 
14 redirector postpones entering the encrypt mode until the 
keys have been generated. While the RSA encryption 
scheme is an effective method for estabhshing a new encryp- 
tion key for each communication session, it cannot be used 
for encrypting the actual data because it requires too much 
computing power to allow on the fly data stream encryption. 
Once the new encryption key for communication session has 

45 been established, the DES stream encryption method is 
employed until the communication session is concluded. 

The technique to transparently encrypt and decrypt 
modem data has also been implemented under Windows™ 
as a Dynamic Link Library ("DLL"). In addition, the DLL 

50 employs RSA public-key cryptography techniques to trans- 
fer per-session encryption key securely to the remote end in 
a manner similar to the TSR implementation. 

Thus, a method and apparatus for providing data encryp- 
tion while using a standard software communications appli- 

55 cation and AT command language compatible modem is 
provided. It will be apparent to one skilled in the art, that 
various embodiments of the invention are possible other 
than the one disclosed. In general, the exemplary embodi- 
ment described herein is merely illustrative of the invention 

60 and should not be taken as limiting the scope of the 
invention. 
What is claimed is: 

1. A method for encrypting text data in a computer system 
running a communication application coupled to a commu- 
65 nication device, comprising: 

a) directing data sent from the communication application 
running on the computer system to the communication 
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device without encrypting said data while an encryp- 
tion application is in a first state, said directing being 
performed by said encryption application running on 
the computer system; 

b) detecting a connect message and a carrier detect signal 
from the communication device while said encryption 
application is in said first state, said connect message 
generated by the communication device in response to 
establishing connection with a remote communication 
device; 

c) changing the state of said encryption application from 
said first state to a remote state when said connect 
message and said carrier detect signal are detected; 
c.l) automatically redirecting the text data to said 

encryption application in response to said encryption 
application changing state to said remote state with- 
out user intervention; 

d) encrypting the redirected data to generate an outgoing 
encrypted data destined to said remote communication 
device when said encryption application is in said 
remote state, said encryption application sending said 
outgoing encrypted data to said communication device; 

e) transmitting said outgoing encrypted data to said com- 
munication device; 

f) detecting a break sequence from the computer system 
or detecting that said carrier detect signal is de- asserted 
when said encryption application is in said remote 
slate; and 

g) changing the state of said encryption application from 
said remote state to said first state when said break 
sequence is detected or said carrier detected signal is 
de- asserted when transmitting said outgoing encryption 
data. 

2. The method as set forth in claim 1 wherein the act of 
encrypting of the redirected data further comprises; 

d.l) decrypting incoming encrypted data sent from the 
communication device to said computer system, said 
encrypted data intercepted by said encryption applica- 
tion prior to said encrypted data being received by the 
communication application. 

3. The method as set forth in claim 2, further comprising: 
asserting a ready/request to send signal if said ready/ 

request to send signal is detected by said computer 
system as being de-asserted when said communication 
device is in said remote state, said assertion of said 
ready/request to send signal preventing a clear to send 
signal from being de -asserted. 

4. The method as set forth in claim 3, wherein the 
encrypting of the redirected data comprises: 

generating a DES data by performing a DES encryption 
operation on an encryption key and an initialization 
data. 

5. The method as set forth in claim 4 further comprising: 
generating said outgoing encrypted data by performing an 

exclusive-or operation on said DES data and said text 
data; and 

generating a next DES data by performing a DES encryp- 
tion operation on said encryption key and said outgoing 
encrypted data. 

6. The method as set forth in claim 1, further comprising 
providing symmetric key stream based encryption methods. 

7. A computer system coupled to a communication device, 
comprising: 

means for detecting a connect message and a carrier 
detect signal from the communication device, said 
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connect message generated by the communication 
device in response to establishing connection with a 
remote communication device; and 
means for automatically encrypting said text data being 

5 sent from a microprocessor application running on said 
computer system to the communication device respon- 
sive to said means for detecting said connect message 
and said carrier delect signal and without user 
intervention, said encrypting being disabled when a 

jQ break sequence is detected or when said carrier detect 
signal is de-asserted from the computer system, said 
encrypting generating an outgoing encrypted data. 

8. The computer system as set forth in claim 7, further 
comprising: 

15 means for decrypting incoming data sent from the com- 
munication device to said microprocessor application. 

9. The computer system as set forth in claim 8, further 
comprising: 

means for detecting the break sequence sent by said 
20 microprocessor application to the communication 
device, said means for detecting said break sequence 
operating while said connection has been established. 

10. The computer system as set forth in claim 9, wherein 
said means for encrypting comprises: 

25 means for generating a DES data by performing a DES 
encryption operation on an encryption key and initial- 
ization data. 

11. The computer system as set forth in claim 10 further 
comprising: 

30 means for generating said outgoing encrypted data by 
performing an exclusive-or operation on said DES data 
and said text data; and 
means for generating a next DES data by performing a 
DES encryption operation on said encryption key and 

35 said outgoing encrypted data. 

12. The computer system as set forth in claim 7, com- 
prising means for symmetric key stream based encryption. 

13. A computer system coupled to a communication 
device compatible with the AT command set comprising: 

a microprocessor running an encryption application, 

said encryption application detecting a connect message 
and a carrier detect signal from said communication 
device when said encryption application is in a first 
state, said connect message generated by said commu- 
nication device in response to establishing communi- 
cation with a remote communication device, 

said encryption application changing from said first state 
to a remote state when said connect message and said 
carrier detect signal is detected, 

said encryption application automatically encrypting said 
text data when in said remote state, said encryption 
application generating an outgoing encrypted data 
without user intervention, 
55 said encryption application detecting a break sequence or 
a de-asserted carrier detect signal from the computer 
system when in said remote state, and 

said encryption application changing from said remote 
state to said first state when a break sequence or said 
do de-asserted carrier detect signal is detected. 

14. The computer system of as set forth in claim 13, 
wherein: 

said encryption application decrypts data sent from the 
communication device to said computer system. 
65 15. The computer system as set forth in claim 14, wherein: 
said encryption application asserting a ready/request to 
send signal if said ready/request to send signal is 
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detected by said computer system as being de- asserted 
when said communication device is in said remote 
slate. 

16. The computer system set forth in claim 15, wherein: 
said encryption application generates a DES by perform- 
ing a DES encryption operation on an encryption key 
and initialization data. 

17. The computer system set forth in claim 16, wherein: 
said encryption application generates said outgoing 

encrypted data by performing an exclusive-or operation 
on said DES data and said plain text data; and 
said CDcryption application generates a DES data stream 
lo be encrypted by performing a DES encryption opera- 
tion on said encryption key and said outgoing 
encrypted data. 

18. A machine readable medium having embodied thereon 
a computer program for processing by a machine having a 
communication device, the computer program comprising: 

a communication application to signal the communication 
device to establish a logical connection with a remote 
communication device; and 

an application to interrupt an operation by the communi- 
cation application, to examine data being loaded into a 
buffer for subsequent transmission over the logical 
connection, to place the machine into an encrypt mode 
when a connect message is detected and a carrier detect 
signal is asserted for encrypting the data before being 
loaded into the buffer, and to place the machine into a 



10 



15 



20 



25 



non-encrypted mode when either the carrier signal is 
de-asserted or a special break sequence is detected. 

19. The machine readable medium as set forth in claim 18, 
wherein the application, being a terminate and stay resident 
program, comprising: 

an interrupt driven serial port driver module to provide an 
Application Program Interface for handling interrupts 
and reading or writing information from a serial port; 

an encryption engine; and 

a redirector module in communication with the interrupt 
driven serial port driver module to receive data from 
the interrupt driven serial port driver and to detiermine 
whether the data is passed through the encryption 
engine to produce encrypted data before being provided 
to the buffer. 

20. The machine readable medium as set forth in claim 19 
wherein the interrupt driven serial port driver prevents a 
deadlock situation, caused when the communication device 
no longer receives data from the communication application 
and the output queue of the communication application is 
full, by asserting a signal allowing all data sent by the 
communication application to be written to the communi- 
cation device when the communication device is in a com- 
mand mode. 

21. The machine readable medium as set forth in claim 18 
wherein the application does not place the machine into the 
encrypt mode if no logical connection is established. 
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